arXiv:1504.01998vl [math.AP] 8 Apr 2015 


A BV Functional and its Relaxation for 
Joint Motion Estimation and Image 
Sequence Recovery 

Sergio Conti, Janusz Ginster, and Martin Rumpf 


Abstract 

The estimation of motion in an image sequence is a fundamental task in image 
processing. Frequently, the image sequence is corrupted by noise and one simulta¬ 
neously asks for the underlying motion field and a restored sequence. In smoothly 
shaded regions of the restored image sequence the brightness constancy assumption 
along motion paths leads to a pointwise differential condition on the motion field. 
At object boundaries which are edge discontinuities both for the image intensity 
and for the motion field this condition is no longer well defined. In this paper a 
total-variation type functional is discussed for joint image restoration and motion 
estimation. This functional turns out not to be lower semicontinuous, and in par¬ 
ticular fine-scale oscillations may appear around edges. By the general theory of 
vector valued BV functionals its relaxation leads to the appearance of a singular 
part of the energy density, which can be determined by the solution of a local min¬ 
imization problem at edges. Based on bounds for the singular part of the energy 
and under appropriate assumptions on the local intensity variation one can exclude 
the existence of microstructures and obtain a model well-suited for simultaneous 
image restoration and motion estimation. Indeed, the relaxed model incorporates 
a generalized variational formulation of the brightness constancy assumption. The 
analytical findings are related to ambiguity problems in motion estimation such as 
the proper distinction between foreground and background motion at object edges. 


1 Introduction 

In computer vision, the accurate computation of motion fields in image sequences— 
frequently called optimal flow estimation—is a long standing problem, which has been 
addressed extensively. For a general overview on optical flow estimation we refer to the 
survey by Fleet and Weiss [22]. We consider an image sequence given via a grey value 
map 

u : (0, T) X n —)■ M ; (t, x) i—)■ u{t, x) 

on a space time domain D := (0,T) x where is a bounded Lipschitz domain in 
for d = 1, 2,3. To begin with, we suppose that motion is reflected by the image sequence 
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and that image points move according to a velocity field v : D ^ Hence, constancy 
of grey values u{t,x{t)) along motion trajectories t i—)• x{t) with x{t) = v{t,x{t)) leads 
to the transport equation 

0 = ®(^)) = dtu{t, x) + Vxu(t, x{t)) ■ v{t, x{t)) (1-1) 

as a constraint equation for the unknown velocity field v. This constraint equation is 
generally known as the brightness constancy constraint and for the space time motion 
field w = (l,u) it can be rewritten as Vu{t,x) ■ w{t,x) = 0. Here and in what follows 
V = {dt,Vx) denotes the space time gradient. This condition gives us pointwise one 
constraint for d unknown velocity components. Indeed, only the component of the 
velocity orthogonal to isolines of the grey value can be computed from equation (1.1), 
which leads to an illposed problem known as the aperture problem. Nagel and Otte 
[37] and Tristanelli [47] suggested to consider second derivatives, i.e. ^Vxu{t,x{t)) = 0 
along motion trajectories 1 1 —)• x{t) leads to dtVxU + = 0 so that, if (V^tt)(t, x) is 

invertible, v{t,x) can be computed. Similarly, in more geometric terms the motion field 
can be described via temporal variations of the shape operator on level sets as proposed 
by Guichard [26]. Since second derivatives are involved, these pointwise approaches are 
vulnerable to noise and hence of difficult practical usability. Based on the assumption 
that the image intensity u varies on a finer scale than v, one might assume v to be locally 
constant and accumulate locally different constraint equations to estimate v. This dates 
back to the early work by Lucas and Kanade [34, 49] or the structure tensor approaeh 
[8], which minimizes the local energy functional u{t — s,x — z){Vu{s, z) ■v{t, x))^ dz ds 
for a local window function w(-, •). 

This paper aims at consistently treating the general case with basically two different 
types of representations of motion in image sequences; 

- mostly smooth motion visible via spatial variations of object shading and texture 
and their transport in time, 

- motion represented by moving object edges, frequently characterized by disconti¬ 
nuities in the motion velocity apparent at edges of moving objects. 

The local approaches mentioned above are able to estimate the first type of motion and 
offer relatively high robustness with respect to noise but in general they do not lead to 
dense flow fields and fail to identify motion information concentrated on edges. Global 
variational approaches were initiated by the work of Horn and Schunck [28]. They 
considered minimizers of the energy functional J^jVu • -|- ajVuj^dxdt implicitly 

assuming the optical flow field v to be smooth. A rigorous numerical analysis for a finite 
difference discretization of the Horn-Schunck approach was performed by Le Tarnec et 
al. [33]. Nagel and Enkelmann [36] proposed to use an anisotropic regularization term 
with a smaller penalization for variations of v in normal direction across edges. With a 
focus on real world applications, Weickert et al. [49] proposed a combination of local flow 
estimation and global variational techniques to combine the benefits of robustness and 
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dense field representation, respectively. For a detailed analysis of the occlusion problem 
associated with the estimation of object motion we refer to the joint approach for motion 
estimation and segmentation by Kanglin and Lorenz [30]. Ito [29] suggested to treat the 
optical flow estimation in terms of an optimal control formulation. Brune et al. [12] used 
the optimal control paradigm to estimate intensity and motion edges in image sequences. 

Before discussing total variation type approaches for motion estimation —to which our 
method belongs— we investigate a basic but already characteristic optical flow problem. 

A simple model problem. Suppose an object O with a shading or texture intensity map 
ui is moving with spatially constant velocity vi on a background with shading and texture 
intensity map U 2 , which is itself moving with constant velocity V 2 - Thus, the observed 
image sequence is given by 

U{t,x) = Xo{x - tVl)ui{x - tVl) + (1 - Xo{x - tVl))u 2 {x - tV2) , (1.2) 

where xo denotes the characteristic function of the object domain O. Trying to retrieve 
the object and the two velocities vi, V 2 from the image sequence one observes the follow¬ 
ing: 

(i) If both image intensities ui and U 2 are constant the role of foreground and back¬ 
ground can be flipped, i.e. either the object O or its complement D\0 is moving 
with speed v = vi , whereas the background velocity V 2 obviously cannot be deter¬ 
mined. 

(a) If both shading or texture intensity maps ui and U 2 are smooth and locally allow the 
computation ofvi and V 2 from (1.1), the decision on foreground and background is 
associated with the consistency of one of the velocities Vi {i = 1,2) with the motion 
of the interface. Thereby, consistency is expressed in terms of the singular coun¬ 
terpart n-Wi = 0 of (1.1), where Wi = (1,^*) and n denotes the space time normal 
on the interface, which coincides with the jump set Ju of the image function u from 
( 1 . 2 ). 

(Hi) If neither ui noru 2 is consistent with the motion of the jump set Ju then the object 
is undergoing a more complex evolution than just a rigid motion, e.g., growth or 
shrinkage. 

In the general case, beyond this simple model problem, at each point on the jump set 
Ju of the space time intensity map u one should compare the values of the velocity v'^ 
and v~ on the two sides with the space time interface normal n on Ju to decide on the 
actual local motion pattern. Hence, we are interested in a variational approach which 
explicitly incorporates this local consistency test, where the motion data v~^ and v~ on 
the two sides are determined either via local shading or texture data or from a global 
relaxation principle taking into account far field motion data. 

The space of functions of bounded variation {BV) allows to describe configurations 
with singularities of codimension one, i.e. edge-type jump sets. Total variation regu¬ 
larization was first introduced in image processing by Rudin, Osher, and Fatemi [45]. 


3 



Cohen [15] proposed to replace the usual quadratic regularization in the above motion 
estimation approaches by a ilC type regularization jVuj dy. A more general convex 
regularizer with linear growth was investigated by Schnorr and Weickert [50]. A broader 
comparison of different regularization techniques in imaging and a discussion of suitable 
quasi-convex functionals was given by Hinterberger et al. [27]. In particular they consid¬ 
ered a W^’P- approximation of BV type functionals. Papenberg et al. [40] investigated 
a TV regularization of the motion field and an optical flow constraint involving higher 
order gradients. In their pioneering paper Chambolle and Pock [14] suggested a dual¬ 
ity approach for nonsmooth convex optimization problems m BV and discussed as one 
application the TV motion estimation problem. An improvement of the original ansatz 
was suggested by Wedel et al. [48] and an efficient implementation of this primal dual 
optimization approach to optical flow estimation was presented recently by Sanchez et 
al. [46]. 

The approach by Aubert and Kornprobst [6] is closely related to ours. They considered 
for d = 2 the energy functional 

= / |Vu • (1, u)|-|-(/>(Vu)-|-dx dt, (1-3) 

Jd 

extended to the space of velocity helds of bounded variation, where u is a fixed image 
intensity map in n SBV and (/> : — )■ M a function with linear growth. They 

discussed the following fundamental problem. On the jump set Ju of u, representing the 
space time edge surfaces, the singular part of the gradient Du is only a Radon measure 
and in the generic case of moving objects one expects a significant overlap of Ju with the 
jump set Ju of the motion field v. Thus, it is unclear how to define Du-w. This is indeed 
a recasting of the above observation (ii) in the context of the theory of functions with 
bounded variation. Aubert and Kornprobst considered a locally averaged evaluation 
of the motion held, and they hnally studied the relaxation of the above functional in 
BV. In the case of Lipschitz continuous image sequences, Aubert et al. [5] considered 
the numerical approximation of the above total variation functional based on a duality 
approach and a suitable approximation of (j) with a sequence of functionals of quadratic 
growth. 

Frequently the image sequence is corrupted by noise that one wishes to remove. On the 
theoretical side, we cannot assume that the input image intensity is already in BV. We 
study here a BV approach for the joint reconstruction of non smooth space time intensity 
maps u and the underlying non smooth optical how helds v. It differs from the ansatz by 
Aubert and Kornprobst [6] in that here both helds are reconstructed simultaneously. We 
investigate in particular the relation to the above-discussed fundamental observations 
(i)—(iii) on motion estimation and show that the BV approach naturally incorporates 
a local analysis of the motion pattern in the vicinity of the jump set Ju. Depending on 
the data, minimizing sequences may develop undesirable small-scale oscillations around 
interfaces. Analytically this means that the functional is not lower semicontinuous. 
The theory of relaxation permits to replace the functional by its lower semicontinuous 
envelope, thereby eliminating the hne-scale oscillations from the kinematics, but still 
incorporating their averaged effect in the energetics. The key technical ingredient we 


4 



use here is a general result on the relaxation of variational problems on vector-valued BV 
functions by Muller and Fonseca [23] and earlier work on relaxation on BV functions by 
Ambrosio and Dal Maso [4] and Aviles and Giga [7]. It leads to the relaxed functional 
presented in (2.3) below. The ambiguity close to the jump set is then resolved by 
minimizing locally a suitable microscopic problem, which in turn leads to a selection of 
the relevant local motion pattern, see (2.4) below. Using upper and lower bounds on 
the relaxed energy we can show that under suitable (implicit) assumptions on the image 
intensity map u and the motion field v such microscopic oscillations can be ruled out. 

The advantages of the joint estimation of intensity u and motion field v are the fol¬ 
lowing; 

- A reliable segmentation of moving objects via the non smooth intensity map helps 
to estimate their motion. 

- Given the motion field, the brightness constancy assumption along motion paths 
significantly improves the denoising of the image sequence or even the restoration 
of missing frames. 

- Reliable motion detection also poses an important cue for object detection and 
recognition. 

Thus, joint approaches which simultaneously estimate the motion field, segment objects 
and denoise the image sequence are particularly appealing. Advances in this direction 
were investigated in [51, 39, 13, 35, 41]. A hrst approach which relates optical flow 
estimation to Mumford-Shah image segmentation was presented by Nesi [38]. Cremers 
and Soatto [17, 18, 19] gave an extension of the Mumford-Shah functional from intensity 
segmentation to motion based segmentation in terms of a probabilistic framework. 

Rathi et al. investigated active contours for joint segmentation and optical flow ex¬ 
traction [42]. Brox et al. [11] presented a Chan-Vese type model for piecewise smooth 
motion extraction. For given fixed image data the decomposition of image sequences 
into regions of homogeneous motion is encoded in a set of level set functions and the 
regularity of the motion fields in these distinct regions is controlled by a total variation 
functional. Indeed, Kornprobst et al. [31] already studied a joint approach for the seg¬ 
mentation of moving objects in front of a still background and the computation of the 
motion velocities. 

The paper is organized as follows. In Section 2 we introduce the variational approach 
via the definition of a suitable energy on space time intensity maps u and motion fields 
V and retrieve the general relaxation result for this type of energies. The functional we 
propose to use is given in (2.3-2.4). On the jump sets of u and v the integrand of the 
relaxed functional involves a microscopic variational problem. The main contribution 
of this paper is to establish bounds for this microscopic energy and to give sufficient 
conditions for the non existence of microscopic oscillations. In Section 3 we discuss the 
consequences of our results for optical flow estimation. Then, in Section 4 we present 
the proofs of the results discussed in Section 2. 


5 



2 Variational approach and relaxation results 


In this section we derive a joint functional for the restoration of a space time image 
sequence and the estimation of the underlying motion field. We start by fixing some 
notation. We denote by Q the image domain, a bounded Lipschitz-domain in with 
d > 1, and hy D = (0,T) x C the associated space time domain. For a 

spatial vector x G we write x = (xi,... ,Xd), while time-space vectors are denoted 
by y = iuo^yi} ■ ■ - yd) £ with t = hq being the time coordinate. Correspondingly, 

the space time gradient reads as V = Vy = {dt, Vx)- We use | • | for the Euclidean norm 
including all matrix spaces. We use standard notation and with 1 < p < oo for 
Lebesgue and Sobolev spaces, respectively. By the fundamental decomposition result 
[3, 21] the derivative of a function of bounded variation / G BV can be written 

as 

Df = VfC^ + D^f -f [/] ® n L Jf 

where (a 0 = aibj for i = 1, • • • , f and j = 1, • • • , /c. Here Jf is the jump set of 

/, [/] = (/■'■ — /“) is the jump, —)• are the approximate limits on Jf, 

n : Jf ^ is the measure theoretic normal to Jf and is the Cantor part of the 
measure Df, orthogonal to both and L Jj. The jump part of Df is denoted by 
D^f = [f]®nn^-^JJf. 

Let uq G U{D) represent the input data, a given grey valued image sequence, with 
p < 1* := Our aim is to determine a restored image sequence tt : D —)■ M and 

an underlying motion field v : D ^ Throughout this paper we use the shortcut 
notation w = (l,u) for the space time motion field. Since we expect that both the 
reconstructed image sequence and the reconstructed image velocities will jump on the 
boundaries of reconstructed moving objects, which are codimension 1 surfaces in space 
time, we consider BV{D) as the suitable space for intensity maps and BV{D;'R.'^) as the 
suitable space for velocity fields. 

We start by defining a functional F measuring the quality of the restoration and 
the motion extraction for an image sequence u G IT^’^(Zl) and a motion field v G 
1F^’^(D;M'^). The actual functional on BV{D) x BV{D;'R.‘^) will then be defined via 
relaxation. For fixed M and ap, a^, ciu > 0 we consider the following energy integrand 
g:M.^ X x ^ [0, oo) 


9 {v,P,q) = aF\w ■ p\j- ay |g| -b |p| 

and define for a general Lipschitz domain U C the energy 

jj^g{v,Vu,Vv)dy if (u, x) G lTbi(t/) x M'^); 

E[u, v,U\ = < ^ 

oo otherwise. 


v\\l°° < M 


( 2 . 1 ) 

This energy is then complemented with a fitting term with respect to the given image 
sequence uq G LP{D) to obtain the functional 

F[u, V, D] = \\u - + E[u, v, D] . 
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The energy E[u,v,D] is finite for {u,v) G x M'^) and ||r'||L°°(D) < M 

with Jjj\Vv\ dy and au J^\Vu\dy measuring the regularity of the motion field v and 
the image sequence u, respectively. Furthermore, 


ap / \w-Vu\dy = aF / \dtu + v ■ Vxu\ dx dt 
Jd Jd 

quantifies the agreement of the pair (tt, v) with the brightness constancy constraint (1.1). 
We remark that for general input image sequences no bounds on the energy do not imply 
a priori bounds on the motion field in L°°, for example in the case that no is spatially 
uniform. The constraint on ||n|| 2 ,°o has been included to avoid this technical difficulty. 

At this point we are ready to define the actual functional of interest for (n, n) G 
BV(D) X BV{D; M'^) as the relaxation F* of the functional F with respect to convergence 
in L^{D): 


I lii 


F* [n, V, D] = inf < liminf F[uk, Vk, D] 
I k^oo 


{uk,Vk) G (n^, n^) ^ (n, n 


( 2 . 2 ) 

As already mentioned in the introduction in the generic case of optical flow applications 
the jump set Jv of the motion field (the union of boundaries of moving objects) is a 
subset of the jump set Ju of the image intensity (the union of all image edges). Hence, 
V is expected to jump on a subset of the support of the jump part D^u = [u] (g) nH^ L Ju 
of the measure Du, so that the term (l,u) • Du is ill-defined. A proper understanding 
of this term requires to select locally a microscopic profile for u and v and includes 
—but will not be restricted to— the proper choice between or v~ for the pointwise 
value for v in the term (l,u) • D^u and thus the local selection between foreground and 
background (cf. the consistency issue (ii) in the simple model problem in Section 1). 
In fact, the theory of relaxation for problems with linear growth is more complex than 
the one with p-growth, p > 1 [20], because of the singular part of the gradient in the 
limit. Relaxation and lower semicontinuity with a convex integrand depending only 
on the gradient field were already obtained in the 60s [25, 43], the general case, with 
dependence of the integrand also on x and u was investigated in the early 90s [4, 7]. 
Here, we use the more general result by Fonseca and Muller in [23], which also includes 
the case of quasiconvex integrands, as the starting point of our investigation. Successive 
developments include [10, 9, 24, 32, 44]. 

These results show that the relaxation F* of F with respect to the L^-topology, as 
defined in (2.2), is finite on the domain 


BV{D) X {BV{D-m.<^) n {u G : \\v\\^^ < M}) 

and it equals 

F* [u, V, D] = IJu - + j^g{v, Vu, Vv) dy + j^g {v, D^u, D%) 

-k f K{u'^, u~, v~^, v~, n) d'H'^. (2-3) 

J j(u,v) 
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Since g is one-homogeneous and convex in the second and third arguments, g coincides 
with its regression function and the third term in (2.3) should be interpreted as 



dD-^u 


dD^v 


d{\D'^u\ + \D^v\)^ d{\D‘^u\ -|- \D'^v\) 


d{\D^u\ + \D%\). 


The key nonconvexity arises from the dependence of g on the first variable in the jump 
part. Thereby, the function K depending on the approximate 

limits («■*■,and {u~,v~) of {u,v) and the measure theoretic normal n on the jump 
set J(^u,v) is the solution of a local minimizing problem in which the energy on the jump 
is optimized with respect to all possible microstructures. Precisely, 


,n) = ml {E[u,v,Qn] '■ {u,v) £ A} , (2.4) 


where Qn is the rotated cube 


Qn — ^ y £ 


\y -n] < \y ■ 


< 


y ■ m 


1 

<2 


with {n, m^,..., rrA'\ denoting an orthonormal basis of and A is the set of 

functions which have traces on the two sides of Qn normal to n and are periodic 

in the m^,..., rrA- directions, 


^ = I (ii, u) G (^Qnj : u = and u = on dQn n |y • n = 

u(y} = u (y + m*) and v(y) = v (y + m*) on 

We remark that the term ||tt — is continuous in u with respect to the weak 

convergence in BV, since we chose p < 1*. Furthermore, both F and F* are coercive in 
BV, in the sense that for any given uq and for any sequence (uj,Vj) with F*[uj,Vj,D] 
bounded, the sequence of BV norms of uj and vj are also bounded. Therefore the above 
representation of the relaxation follows immediately from the more general statement 
of Fonseca and Muller [23, Theorem 2.16] and existence of minimizers for the relaxed 
functional F* follows easily by the direct method of the calculus of variations. 

A practical usage of the functional F* requires knowledge of the effective surface 
energy K, much as in the case of relaxation on spaces one needs to determine 

the quasiconvex envelope of the integrand [20]. A numerical computation is in principle 
feasible via the minimization in (2.4), but it is nevertheless useful to extract analytical 
information on K as far as possible. In what follows we give lower and upper bounds for 
the singular term K and compute it explicitly in special cases. Thereby, we show that the 
model favors locally simple (i.e. planar) profiles in the microscopic problem (2.4) under 
reasonable assumptions from the viewpoint of the practical optical flow application. This 
renders the functional F* well-suited for joint image sequence restoration and motion 
extraction. A detailed discussion of the consequences of the bounds for K is postponed 
to Section 3. 





Theorem 2.1. Givenu^,u ^'K,v~^,v ^and n ^ S'^ for K = K {u'^, u ,v~^,v ,n) 
the following statements hold: 

(i) If {w^ ■ n) {w~ ■ n) < 0 then K = | [u] | + a^, | [v] | , 

fa) For d = 1 and |[«1| < one has min K(N^) < K < min K(N~^), where 

K{N'^) = (a„ |[tt]| + Ot, INI) + |^~|) + ttF IMI {\N'^ ■ + \N~ • , 

K{N~^) = (a„ |[u]| + ay |[r]|) (jiV^j + j^"!) + IMI \N~^ • t(;+ + • w”| , 

with N~ = n — N~^, 

(Hi) For general d>l one has K < Kd, where 


Kd = min 



+ a„K-.-^|+o„|.--,^|)|JV^|+aHMlE \N^- 

i=i 


w' 


where and the minimum is taken over I £ N and the set of vectors 

v^,... ,v'' £W^ and N^,.. .N^ £ subject to = ^• 

The proof is given in Section 4.3. 

The upper bounds in Theorem 2.1 are based on suitable choices for the optimal mi¬ 
croscopic solution u, V of (2.4). If the profile of these microscopic solutions u,v depend 
only on y • n, we call it simple. There are in particular two types of simple profiles when 
solving (2.4) for given data {u~,v~), (u+jU"''), and n. In the case (re"'' • n) {w~ • n) < 0 
we define the following piecewise constant functions u £ BV{Qn) and v £ BV{Qn;^'^). 


u{y) = 



if y • n < 0 
if y • n > 0 


and v{y) 


V 



if y • n < — ^ 

if - I < y ■ n < I , 

if y • n > I 


(2.5) 


where v^ is chosen in the line segment [v , u'*'] such that - n = 0 for = (1, v^). For 
suitable approximations in such as those obtained by convolution Uk = ^X{\y.n\<^* 

u, Vk = |X{|y.„|< 1} *v we have that 


K{u'^,u ,v~^,v ,n) < liminfF;[uA;,Ufc,(5fc] = a^lMI+an|NI , (2.6) 

k^oo 


where [u] = v~^ — v~ and [tt] = — u~. This profile corresponds to a microscopic 

consistency with a BV interpretation of the brightness constancy constraint (1.1). In 
the other case {w~^ ■ n) {w~ • n) > 0 consistency of a simple and optimal microscopic 
profile is out of reach. Let us assume that \w~ ■ n\ < Ire"*" • n\. Then a feasible, simple 
profile for the minimization of E[-,-, Qn] is given by 


u{y) 



if y • n < 0 
if y • n > 0 


and u(y) 


V if y • n < ^ 

v^ if y • u > g 
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Figure 1: A sketch of the two simple profiles for {w~^ ■ n) {w~ • n) < 0 (left) and 
{w~^ ■ n) {w~ ■ n) > 0 with |rt;“ • n| < Itc'*' • n| (right). The black line indi¬ 
cates the jump set of v. In the dark grey area u takes the value u~, on the 
light grey area the value u~^. 


Indeed, again using the same ansatz for the approximation as above, one obtains 

K{u~^,u~,v~^,v~,n) < liminf E[uk,Vk, Qk] = OiF\w~ • n| -|- [u]| -|- [«]| • (2-7) 

k^oo 

Figure 1 shows sketches of the simple profiles in both cases. In the next theorem we 
establish sufficient conditions for the existence of minimizing profiles which are simple. 

Theorem 2.2. Let u'^,u~ G M, v~^,v~ G and n G 5'^, then we have the following. 

(i) If {w^ ■ n) {w~ • n) < 0 then the optimal profile for K is simple and 

K = a^lHI -h a«|M| . 

(a) Ifd=l, {w~ ■n){w^ -n) > 0, |[n]| < ^ and 2 {au\[u] \ + a„|H|) |ni| > ofIMHNI 
then the minimizing profile for K is simple and 

K = ai?min{|rc“ • n|, • n|}|[tt]| -|- aD|[u]| -|- a,i|[u]| . (2-8) 

(Hi) If d> 1, {w~ ■ n){w~^ • n) > 0 and |[ii]| < AhliF then the minimizing profile for 
K is simple and given by (2.8) . 

(iv) If the profile is simple, then 

K = min (uu |[w]| -k lu"'' —a| -|- |u“ —a| -|- ap |M| \n • (1, a)|) . 

The proof is given in Section 4.3. An example where the profile is not simple is 
discussed in Section 3 below. 
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3 Consequences for the motion estimation 


In this section we discuss the practical implications of Theorem 2.2 on the actual optical 
flow estimation based on the proposed variational approach for the joint image sequence 
restoration and motion extraction. We will analyze implicit conditions on the approxi¬ 
mate limits of u and v at jump sets under which the singular energy density K defined in 
(2.4) is simple and there is no relevant microscopic scale arising in the variational model. 
Furthermore, we will give an example where microstructures actually appear and thus 
the singular energy density K is not simple. As the only interesting cases appear where 
[u] / 0, in this section we always assume that Jv C Ju- 
We start observing that the condition for simple profiles in one space dimension in 
Theorem 2.2 (ii) includes in particular the case 

IMI<2"^|ni| , (3.1) 

Otp 

as well as the case 

cv ry 

IMI < 2— Iml and |[u]| < 2— . (3.2) 

ctp ap 

Hence, for moderate speed of the intensity interface Ju (associated with large ni) and 
either moderate difference |[u]| of the estimated motion on both sides of the interface 
or moderate intensity variation |[u]| the condition in Theorem 2.2 (ii) is fulfilled (in 
particular for (|ni| > ^ and > ||u||i,oo) or (^ > ||u||loo and \ni\ > ^^^^)). For the 
case d > 1 the condition in Theorem 2.2 (iii) can be rephrased as 

IH-(ni,....n,i)l>^l|!>IIIH|. 

lOiy 


Thus, if ^ is small, this condition is fulfilled for a moderate speed of the intensity 
interface Ju (associated with large spatial component of n) and for a direction of the 
jump ||(4 t with a significant component pointing in direction of the spatial interface 


normal (^1 

normal • 

In the practical application the velocities on the two sides of the discontinuity u"*" and 
v~ are determined within the variational setting by shading or texture information on 
both sides of the edge set Ju- Then, the singular energy density K{u'^,u~,v~^,v~,n) 
is associated with the proper identification of the type of object motion as described 
for the simple model problem in Section 1, in particular the decision on foreground 
or background and the identification of additional erosion or dilation. As we will see 
below, for K simple the minimization of the joint functional is able to decide on the 
motion pattern. In the case that K is not simple and that microstructures appear 
the variational model seems not to appropriately reflect the scope of possible motion 
patterns. On the other hand, under the reasonable implicit assumption that the data 
u~^, u~, u'*', v~ fulfills one of the conditions under which the singular energy density 
K is simple the joint variational approach actually renders the coupled restoration and 
motion estimation problem meaningful including the proper identification of the local 
motion pattern at object edges. 
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Figure 2: ID Sketch of different configurations at the interface Ju of a moving object 
{u = u'^) with motion velocity v~^ on a background with motion field v~ and 
On the left the interface motion is consistent with the object 
motion, in the middle the object is eroding and on the right the object is 
dilating. 


In what follows we study the different local motion pattern at some point y £ Ju and 
the associated singular energy density K in more detail. 

Consistent interface motion. 

Let us suppose that a light object with image intensity u'^ is moving with a speed v'^ on 
a dark background with speed v~ and image intensity u~ (m"'' > u~) (cf. left sketch in 
Fig. 2). The consistency of the interface motion of the space time edge set Ju with the 
object motion is expressed in terms of the brightness constancy assumption • n = 0 at 
the point y £ Ju with n denoting the space time normal on Ju, i.e. Du = [u]'S)n'H'^JJu- 
In this case Theorem 2.2 (i) applies and K = a,;|[u]| + a„|[ii]|. Microscopically this is 
realized by the profile sketched in Fig. 1 (left) with = v~^. Due to noise the brightness 
constancy assumptions might only be approximately fulfilled with Itc'*' • n| <C 1. Then, 
Theorem 2.2 (ii) shows that we obtain the corresponding approximate singular energy 
density K = aF\w^ • ^IIMI + ®'y|['^]| + «u|MI- 

Non consistent interface motion. 

Let us suppose that in the same configuration w~ ■ n < w~^ • n < 0 (cf. middle sketch 
in Fig. 2). Hence, neither the object motion nor the motion v~ (currently classified 
as background) is consistent with the motion of the interface Ju- Indeed, we observe 
an erosion of the interface. Let us assume that v is the actual speed of the interface 
with w ■ n = 0 for w = (l,u), then v — v~^ is the effective erosion velocity and |(0, u — 
u"*") • n\ = Ire'*' • n\ is the associated footprint in the singular energy density K, which 
is in agreement with the findings of Theorem 2.2 (ii). Obviously, the energy functional 
considers this classification as favorable compared to the one obtained via flipping object 
and background and classifying a foreground object with intensity u~ moving with speed 
v~ and dilation speed v~ —v and larger footprint in the singular energy density \w~ • n|. 
Microscopically this is realized by the profile sketched in Fig. 1 (right). If w~ • n < 0 < 
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■ n (cf. right sketch in Fig. 2) the variational approach favors the classification of a 
dilation process on the interface Ju with velocity v — v~^ for the velocity v on the line 
segment [fwith w ■ n = (1,1;) • n = 0 and thus smallest possible singular energy 
density K = Q:.y|[u]| +au|[u]|. The associated microscopic solution profile coincides with 
that sketched in Fig. 1 (left). 

Objects and background with constant intensity. 

Objects and background with constant in¬ 
tensity. If there is no shading or texture 
information, the estimation of motion ve¬ 
locities can solely be based on the ob¬ 
served motion of interfaces. Examplarily, 
let us suppose that objects Oi,, Om are 
moving with constant velocities ui,..., Um 
in front of a immobile background as 
sketched in Figure 3. Furthermore, let us 
explicitly rule out interface dilation and 
erosion. For such a configuration the sin¬ 
gular energy density compares the differ¬ 
ent foreground and background configura¬ 
tions in the local depth ordering. For a 
hxed depth ordering of the objects the estimation of a velocity Vi for each object is 
based on the minimization of fg^yis K with 

K = q;f|(1, Vi) ■ n\ + - v°^^\ -|- , 

where is the visible part of dOi and are the velocity and the intensity 

opposite dOi, respectively. The estimation of the depth ordering via the variational 
approach is then performed by a comparison of the minimal total energy obtained for 
the set of all possible depth configurations (In Figure 3 the color of the interfaces shows 
which is the relevant velocity of the singular energy density for the minimal total energy). 
An estimation of the background speed is obviously not possible. 

Appearance of microstructures. 

By the above discussion it is clear that we have to deal with large relative velocities 
and jumps in the intensities. The key idea is to find a case where the upper bound in 
Theorem 2.1(ii) and (iii) is lower than the energy of simple profiles given in Theorem 
2.2(iv). 

We discuss first the case d = 1. Let n £ and let K be the upper bound given in 
Theorem 2.1(ii). We assume that 

I [u] I < and 0 < w~ ■ n < w~^ ■ n . 

ap 



Figure 3: Overlaying motion. 
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Then the energy of a simple profile in Theorem 2.2(iv) coincides with K{0). Indeed, 
if a is between v~ and then |ri+ — a| + \v~ — a| > — v~\ and |n • (l,a)| > 

|n • if a is outside that interval then, letting b be the projection of a on the interval. 
If"'' — a| + |r;“ — a| = 1^^ — r;“| + 2|a — 6| and \n ■ (1, a)| > \n ■ b\ — \b — a\. This proves 
that the minimum is attained at a = 

Therefore it suffices to construct a situation where 0 is not a minimizer of the upper 
bound function K. For a vector G chosen later and a small <5 G (0,1) we compute, 
assuming n • w~ / 0 and writing for brevity C = ^ 

+ |n - <5Ar| - 1 + C (|(jiV • u;+| + |(n - 6N) ■ vj-] - \n ■ u;-|) 

au\[u\\ + av\[v\\ 

= A{N)6 + 0{6^) 


where 

A{N) = \N\-n-N + C(|A^ • u;+| - iV • u;“). 

Therefore it suffices to show that G can be chosen so that A{N) < 0. 

We choose a large velocity u > 1, set ap = = Ou = I, = ±u, u'^ = l,u~ = 0, 
so that [u] = 1 < ‘la^jap^ C = 1/(1 + 2u), = (l,±u). We choose a normal which 

corresponds to motion with a velocity close to but not identical with the velocity v~. 
Precisely, for some v > v we set n = {v, l)/\/u^ + 1. Then it is easy to see that 
0 < n • w~ < n ■ w~^. Finally we set N = (u, — 1), so that N ■ = 0. It remains to show 

that V and v can be chosen so that A{N) < 0. To do this we compute 


A{N) = |A^| - n-A^ + C(|A^-u^+ 


N-W-) 



2v 

l + 2v 


which is negative if v and v are chosen sufficiently large (cf. Fig. 4). A detailed 
computation shows that v = 2 and any u > 4 will do. In the case v = v, however, the 
Taylor series above is not admissible (since w~ • n = 0) and the profile becomes simple 
again, in agreement with Theorem 2.2(i). 

The construction can be easily generalized to the case d > 1. From the lower bound 
in Theorem 2.2(iv) we know that the best interfacial energy which can be attained using 
simple profiles is given by 


Kg = au\[u]\ + — a + V —a + Oi? |[u]| |n • (l,a)| 


for some a G We choose as above a large velocity v > 1, set ap = ctv = ctu = ^, 
= ±uei, = (I,u^), = ±1, and pick a normal which corresponds to motion 

with velocity —vei, namely, n = , 0)/\/l + i;^, for some v > v. The optimal a 

is also parallel to ei, and a short computation shows that it equals so that 


Ks = au\[u\\+av V — + ap\[u]\\n ■ w 


The one-dimensional result shows that there is N~^ G such that K{N^) < Kg. This 
result can be immediately embedded in the higher- dimensional setting by taking I = 2, 
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Figure 4: Sketch of a situation where microstructures develop. On the left the ID sketch 
of a moving object with relatively high velocity in space-time. On the right a 
microstructure profile that has a lower energy than a simple profile. 

= {N~^, 0,..., 0), N'^ = n — N^, = v~^, v'^ = v~ in the upper bound of Theorem 

2.1(iii). Therefore the same values, v = 2 and v > 4, will do. 

We refer to Section 4.4 for a visualization of a more general microscopic pattern in 
the case d = 2. 

Potential impact on the numerical implementation of the motion estimation model. 

The model studied here has a built-in consistency with respect to the local shading or 
texture information and to the global geometry of moving and deforming objects in 
space-time, and is therefore attractive for concrete applications to imaging, based on an 
appropriate numerical implementation. The possible appearance of microstructures is, 
however, a very problematic feature of the model. If microstructure appears the usability 
of the model in imaging is questionable. 

The analysis presented in this paper shows that microscopic patterns appear only 
if the motion velocity v encoded in the edge set Ju is relatively large and the motion 
data v~^ and v~ encoded in the shading or the texture on both sides of the edge are 
substantially inconsistent with v. Thus, in imaging applications the criteria for simple 
profiles stated in Theorem 2.2 and discussed at the beginning of this section mostly rule 
out the appearance of microstructures and indicate that a one-scale method should be 
appropriate in normal situations. The criteria in Theorem 2.2 can be used in a numerical 
algorithm as an a-posteriori test for the appropriateness of the one-scale model based on 
upscaled versions of ^u~ ,v~, and n. If this test fails, it may be advisable to modify 

the the parameters a„, and ap, in ways which are suggested by the conditions in 
Theorem 2.2. Failure of the test, if one does not appropriately modify the parameters of 
the problem, leads to microscopic patterns. In particular, one would expect to observe 
oscillations in the single-scale numerical approximation of u and v in the vicinity of 
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the edge set Ju, with a length scale given by the spatial discretization. In fact, the 
constructions used in the proofs of the upper bounds could be used to set up a reduced 
microscopic model in a two-scale discretization approach, which would include a local 
optimization over a small set of parameters describing the microstructure. Numerical 
two-scale methods of this type are of theoretical interest, but in this concrete application 
they are of no practical relevance. 

For a practical implementation of the single-scale model it is useful to observe that 
the functional F is not jointly convex in u and v. However, F is separately convex 
in u and in v and thus classical primal-dual methods can be applied in an alternating 
minimization scheme. 

4 Proofs of the main results 

In this Section we prove Theorem 2.1 and Theorem 2.2. We start by discussing the 
symmetries of the function K and reformulating its definition on a different space of 
test functions in Section 4.1. Moreover, we prove classical subadditivity and convexity 
properties for K. Then, in Section 4.2 we discuss explicit constructions which lead to 
upper bounds of iF for d = 1 and prove a lower bound of IF for d = 1. Based on 
the ingredients of Section 4.1 the proofs of the theorems are then given in Section 4.3. 
Finally, in Section 4.4 we illustrate the possible microscopic patterns which might arise 
in critical regimes complementing the discussion in Section 3. 

4.1 Preliminaries 

In order to simplify the following discussion we first list the symmetries which K obeys. 
Lemma 4.1. K has the following symmetries: 

K (tt+, ,v~ ,n) = K (u+ — u~ ,v~ ^n), 

K (tt+, v^,v~,n) = K v~,n), 

K {u^,u~, v^,v~,n) = K {u~,u'^,v~,v~^,n), 

K {u^,u~, v~^,v~ ,n) = K {u~, u'^, v~, v~^, —n). 

Proof. They all follow immediately from the definition. For the hrst equality replace 
u hy u — u~. For the second item replace u by —u. For the third equality replace u 
and V by u{—y) and v{—y). The last invariance is simply a relabeling of the boundary 
conditions. □ 

One key observation is that we can replace the set of functions in the definition of K 
by a simpler class of functions. In particular, the function u can be assumed to take 
only two values, and the velocity held v can be assumed to be smooth. This allows to 
give a classical sense to the term w ■ Du. We stress that existence of minimizers is not 
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expected in this restricted class, indeed functions of this class are later interpreted as 
microstructures. 

Lemma 4.2. In the definition of K in (2-4) the set A can be replaced by 

= 1(^,1;) G BV [Qn] X : u = and v = on 

dQn periodic in the -directions 

Proof. We prove the claim in three steps. If [tt] = 0 then one immediately obtains 
K = a„|[u]| from which the result follows easily by standard density results. Hence we 
only need to deal with the case [u] 0. 

Step 1. First we show that miE[A°°,Qrf[ < miE[A,Qrf[, where 

^OO ^ ^ n A 

To prove the inequality, we choose {u,v) G A and can assume that ||u||loo < M. We 
extend u and v periodically to |y G : \y ■ n\ < and constantly in the n-direction 
and define {u,v) iu) = {u,v){2y). Then a straightforward computation shows that 

E[u,v,Qn] = E[u,v,Qn ]. 

For 0 < e < j we define (u^jVs) = (u,v) * Pe where pe G is a standard 

mollifier. Since iu,v) is constant on |y G : y ■n>\] and {y G : y • n < — j}, 

the functions {us,Vs) satisfy (ue,Ue) G A°°. 

To prove convergence of the energy we first observe that by the general properties 
of mollification of Sobolev functions we immediately obtain —)■ u and Ue —)■ u in 

(Qn', I^) and {Qn, 1^^) > respectively. The more subtle term is fg \w£ ■ Vu^l dy, 
where = (1,U£). Since llrcell^^oo < ||th||x,cxj < M + 1 we can estimate 

llu;^ • < Ihe • (Vue - ^ u )\\ l ^ q „) + + 11^ • • 

The first term on the right hand side can be estimated by 

llrce • (Vu^ - < \\ w \\ l ^{ q „) llVue - ^ 0. 

For the second term we observe that rcg — rZJ —)■ 0 pointwise a.e. and that 

|(tc£ — w) ■ Vu| < 2 ||ta|lLoo(Q^) |Vu| pointwise. 

Hence by Lebesgue’s dominated convergence theorem we derive 

lini ||(rc£ -uJ) • = 0. 
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Therefore 


mlE[A^,Qn] < limsup^[rte,r>£,Qn] < E[u,v,Qn] = E[u,v,Qn] 

£—^0 

concludes the proof of the first step. 

Step 2. We show that mi E[A^,Qn\ < mi E[A°°,Qn\- Let {u,v) G A°°, assume for 
definiteness that uA >u~. The coarea formula yields 



{ou |Vm| + aplw ■ Vtt|) dy > 




otu \Dx{u<t} \ +aF\w- Dx{u<t}\^ 


dt. 


Indeed, the first term is standard, the second one follows easily from [3, (3.33) in Th. 
3.40] approximating w uniformly with piecewise constant vector fields. 

Thus there exists t* G such that 


/ {au\Vu\+ aplw -Vul) dy > {u'^ - u ) / au\Dx{u<t*}\ + oif\ w ■ Dx{u<t*} 

'^Qn Qn 


We define = {y £ Qn '■ u{y) < t*} and u* = + {u — u~^) Xq- ■ Then 


/ au jVuj + aF 

Qn 

where Du* on the right hand side has to be interpreted as a measure. Since the function 
w is smooth, it is in particular continuous on the jump set Ju*, hence the integral is well 
defined. Furthermore by the trace theorem u* fulfills the boundary conditions. Therefore 
{u*,v) G A^ and inf E[A'", Qn] < inf E[A°°, Qn]- 
Step 3. We prove that inf£'[^, < miE[A^ ,Qn]- As in Step 1 we choose (rt, u) G 

A^, extend both functions periodically, scale them to {u,v): nnd mollify u (but not v) 
to obtain = u * pe. For the same reasons as in Step 1 we have {us,v) ^ nnd 

lim / |Vrte|dy= / \Du\ = / \Du\ . 

jQn JQn JQn 


jrc • Viij dy> au]Du*\ + uf ]w - Du* 


' Qn 


It remains to show that Jg \w ■ Vue] dy —^/q ]w ■ Du\, where as usual w = (1,F). To 
see this, we deduce from \Du\{dQn) = 0 for the Radon measure w ■ Du that (cf. [3, 
Prop. 3.7]) 

lim / \ps * {w ■ Du)\dy = / \w ■ Du\ . 
jQn JQn 

Furthermore, 



[pe * {w • Du) 


w ■ {pe* Du)] dy 



Pe{y - z){w{z) 


w{y)) ■ Du{z) 


dy 


converges to zero because w is uniformly continuous on Qn- Hence 


\imsup E[Ue,V,Qn] < E[u,v,Qn] = E[u,v,Qn] , 

£—^0 

which proves the claim. □ 
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Next we give an iterated relaxation formula. 

Lemma 4 . 3 . For (tt, v) G we have 

K{u'^, u~ ,v'^,v~ ,n) = inf {S* [u, v, Qn] '■ {u, v) G A'^} , 

where E*[u,v,Qn] = Jq av\Dv\ + Jq K{u'^,u~, v,v,u)dT-L'^ for {u,v) G A'^ with v 

being the normal to Ju- 

Proof. Let E* be the relaxation of E, defined as in (2.2). From the general relaxation 
result we know that it has a form corresponding to (2.3), namely, 

E*[u,v,Qn] = I g{v,\7u,Vv) dy + f g{v,D^u,Vv) 

+ f K [u'^ ,u~ ,v^ ,v~ , 1 /) d'H'^ . 

^(u,v) ^Qn 

If {u,v) G A^ in particular Fi* [u, u, Qn] = Jq av\Dv\+ fg K(u~^,u~ ,v,v, n)d?{'^, as 
given in the statement. 

Fix u~, v~, n; let (u,v) G A^, extend u,v periodically and dehne as in 

Step 1 of the proof of Lemma 4.2 {uj,Vj) = {u,v){jx). Then {uj,Vj) —)• (u*,u*) = 
{u~,v~) + ([u], [v\)Xy-n>Q in L^. Hence, by the lower semicontinuity of E* we obtain 

K{u'^,u~,v~^,v~,n) = E*[u^,v^,Qn\ < IrmmdE*[uj,Vj,Qn] = E*[u,v,Qn] ■ 

j->oo 

The other inequality follows immediately from Lemma 4.2 and E > E*. □ 

Next we prove two more general properties, namely, subadditivity and convexity of 
the surface energy K, which are well known for example in the setting of variational 
problems on partitions [1, 2]. We start with subadditivity and its main consequence in 
the present setting. 

Lemma 4.4 (Subadditivity). Let u~^,u~ G M, v~^,v~ G n G Then: 

(i) For any u' G M, u' G one has 

K{u^ ,u~, v~^ ,v~ ,n) < K{u~^ ,u, v'^, v', n) + K{u , u~ ,v', v~ ,n). 

(a) For any a G one has 

K{u'^,u~, ,v~,n) < a^l[u]| + a„(|u^ — a| + |u“ — a|) + ai?![tt]| |(1, a) • n| . 

Proof, (i): For j G N we define 


{uj,Vj){y) 


( (u+,U+) 


l(« ) 


if 2/ • n > j 

if -i<yn<i . 

if 2/ • n < - j 
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Clearly {uj,Vj) (woc^oo) = {u ,v ) + {[u\,[v])xyn>o in and E*[uj,Vj,Qn] = 
K{u~^, u' , v^,v' , n) + K{u', u~, v', v~ , n) for all j > 2. Since E* is lower semicontinuous, 
we obtain 


K{u'^,U ,V~^,V ,n) = E*[Uoo,Voo,Qn] E*[uj,Vj,Qn] , 

j^oo 

which concludes the proof. 

(h): Two applications of (i) and the fact that K = a«|[f]| if IMI =0 give 

K{u'^, u~ ,v^,v~,n) < K{u~^,u'^,v~^,a, n) + K{u~^,u~ ,a, a, n) + K{u~,u~ ,a, v~, n) 
< av\v^ - a| + (anIMI + ofIMI 1 ( 1 ,«) • n\) + av\a - r;“| . 


□ 


We next show that K is convex in the last argument, after having been extended to 
a positively one-homogeneous function. This will be linked to the possible development 
of oscillations of the interface. 


Lemma 4.5 (Convexity). For given u~^,u G M, v~^, 
by 

h{n) = \n\K{u~^ ,u~ ,v~^ ,v~ 


V ^ we define 



h : M 


The funetion h is positively one-homogeneous and eonvex, in particular, 


h{n'^ + n )<h{n'^) + h{n ) foralln'^,n G (4.1) 


Proof. Positive one-homogeneity is obvious from the definition. It implies Xh{n^) -\- 
(1 — X)h{n~) = h{Xn'^) -|- h{{l — X)n~), hence convexity is equivalent to subadditivity; 
therefore it suffices to prove the inequality (4.1). Furthermore, it suffices to consider the 
case that |n+ -|-n“| = 1 with n~^ and n~ linearly independent (the case that n'^ and n~ 
are parallel, including the case that one vanishes, is immediate). 

By the scaling argument used in the proof of Lemma 4.3 it suffices to construct 
functions {u, v) : —)• which are periodic in the d directions orthogonal to 

n = n"*" -|- n~ and with {u,v){y) = {u^,v^) whenever ±y ■ n > L, for some L > 0. We 
shall first construct a curve 7 that can be interpreted as the jump set of a function u 
satisfying the boundary conditions. The curve 7 will consist of two subsets, one has 
normal and measure the other has normal n~/\n~\ and measure \n~\. To 

make this precise, we define = Jn^, where J G SO{d -|- 1) is a 90-degree rotation in 
the plane spanned by n+ and n~. We define 


7 '*' = [ 0 , l)np and 7 = Up [ 0 , l)np , 

so that 7“ ends at n+ -|- n“ = Jn, with n = n"'' -|- n“. We set = Jn, choose 
w ?,..., such that {n, m^,..., is an orthonormal basis of and define S = 

Zm^ -|- (7+ U 7“) -|- Ylii>2 see Fig. 5 (notation: ^ -|- cB = {a -|- c 6 : a G vl, 6 G B}). 
This is a d-dimensional surface, corresponding to the constant extension in the directions 
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Figure 5: The construction of 7 is a periodic laminate nsing 7 + and 7 “ that splits 

into two connected components 0 ;"*“ and oj~. Then u is defined as u~ on uj~ 
and as u'^ on a;+. 


of the cnrve obtained joining alternatively copies of the segments 7 '*' and 
7 “. In particnlar, \ S has exactly two connected components; one of them, call 

it u}~, contains {y ■ n < —L} and the other, call it , contains {y ■ n > L}, where 
L = |n+| + \n~\ + 1, see Fig. 5. We set {u,v) = {u~,v~) + ([n], [n])x^+. Then the 
normal to the jump set is on a set of measure In^ | = and correspondingly 

for n~. We define by rescaling {uj,Vj){y) = {u,v){jy), so that {uj,Vj) —)■ = 

{u~,v~) + ([n], [v\)xy-n>o in . Therefore, dropping the arguments ,u~,v~ for 
brevity, 

E*[u^,v^,Qn] = h{n^ + n~) = K{n^ + n~) 

71 

< \ri^\K{——) + |n“|iF(^-r) = h{n^) + h{n~). 

n+ n 


□ 


4.2 Upper and lower bounds for c? = 1 

Proposition 4.6 (Construction for d = 1). Let d = 1 and u'^ , u~ , v~^, v~ G M, n G S^. 
Then for any N~^ G we have 

K{u'^, u~ ,v~^,v~,n)< (Q;„|[n]|+Q:.„|[n]|)(|7V''| + |A^“|) Tap |[n]| {\N^- rc’''| + \ N~- rc“|) , 
where N~ = n — N~^ and = (l,n^). 

Proof. The result follows by Lemma 4.5 and Lemma 4.4(ii), taking once a = and once 
a = v~. For the sake of illustration we give here a self-contained, explicit construction. 


21 




We start by replicating the construction of Lemma 4.5. Again, we first construct a 
curve 7 that will be interpreted as the jump set of a function u satisfying the boundary 
conditions, and which consists of two subsets, with the normals A""*"/! and A^“/|A^“|. 
Precisely, we define as in the proof of Lemma 4.5 

7 += [ 0 , l)(7V'')-‘- and 7 “ = + [ 0 , l)(7V“)-‘-, 

so that 7“ ends at {N^ + = n-*“. We set 7 = n-*“Z + (7+ U 7“) and L = 

|A^'*“| + |A^“| + 1 . Then \7 has exactly two connected components and for L sufficiently 
large one of them, call it uj~, contains {y ■ n < —L} and the other, call it contains 
{y ■ n > L}, see Figure 5. We set u = u~ + {u'^ — u~)Xi^+- Then u fulfills the desired 
boundary conditions. 

Ideally the function v should equal v~^ on a;+u( 7 ''' + n-^ Z),andu oncj u (7 +n-‘-Z). 
As an approximation we construct a function as follows (cf. Figure 6 ). Precisely, for 
e G ( 0 , 1 ), we set 

Lof = w"*" U + n'''Z) \ + n'''Z) 

and Ve = v~ + ~'^~)Xco+- It is easy to see that {DveKQn) < I'C"'' — + |A^“| + 

dvre). Let now G C°°, where (ps G C^{B^) is a mollification kernel. Then 

(cf. Figure 6 ) there exists a constant c, such that 

(i) Ve = v~ on 7 “; ^^^( 7 + \ {v^ = u+}) < ce, 

- IMI 

(hi) Vs = on {±y • n > L}. 

Thus (tt, Ve) G and a straightforward computation shows that 

E[u,Ve,Qn\ =(aM|MI +a«IMI) {\^~^\ + \^~\ +ce) 

+ aj? |[u]| (IA"^ • + |u;“ • A^“| + c|[u]|e) . 

Finally, we can take e arbitrarily small, which establishes the claim. 

□ 


In order to prove the lower bound in Theorem 2.1 (ii) we need the following truncation 
lemma. As it is frequently the case in truncation, this can only be done if the relevant 
field (the velocity here) is scalar, and the result is therefore restricted to d = 1 . 

Lemma 4.7 (Truncation for d = 1). Let d = 1 and u'^,u~,v^,v~ G M 6e such that 
v~ < v~^ . Let n G S^, {u,v) G A^ and v : Qn -G be defined as 


v{y) 


v{y) ifv <v{y)<v+, 

< v~^ if < v{y ), 

ifv{y)<v-. 
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Figure 6 : The picture shows the construction of 
u and V. On the dark part u takes the 
value u~, on the grey part the value 
The brown line is the part of the 
jump set of u parallel to N~ (7“), 
the blue line represents the part of 7 
with normal parallel to N~^ ( 7 '*")- The 
black line indicates up to smoothing 
the jump set of v^- As can be seen 
this is in terms of length and normal 
essentially the jump set of u. 


-(f IMI < 2^, then we have 


tty / iVul dy + ap / w ■ Du > / [Vul dy + ap / w-Du, (4.2) 

'^Qn Qn '^Qn '^Qn 

where w = ( 1 , v) and w = (l,u). 

Proof. Let Q~ = {y G Qn : u{y) = u“}, 7 = {dQ~) n Qn and n'^ be the outer normal to 
Qf on 7 , so that Du = [u] ( 8 > n'^d'H^^j. We want to estimate the term 

— w) ■ n'^ dPL^ . 


ap / {w — w) ■ Du = ap[u] / {w — w) 
Qn 'y 


Since w — w vanishes on {y ■ n = —1/2} and it is periodic in the direction we have 
that — w) ■ p dT-L^ = 0 (here p is the outer normal to Q^), and the same on 

Qn — Qn \ Qn ■ Therefore by the Gauss-Green formula 

/ {w — w) ■ n'^ dPL^ = / div(u; — w) dy = — / div(t(; — w) dy . 

■^7 -tQ- Jq+ 

Estimating | divtcj < |Vu| and averaging over the two sides of 7 we obtain 

Ml 

|Uiv ytc — w)\ wy ^ up- 

t Qn ^ JQn 


a p 


{w — w) ■ Du < ap^\j^ [ I div (tc — tc) I dy < f |Vu — Vu| dy 

2 JQn 2 Jq^ 


= Oip- 


Ml 


([ |Vu| dy- / |Vu| dy) 
\dQn JQu / 


<ay |Vu| dy-a^ / |Vu| dy, 

'^Qn Qn 

where we used that locally either v = v and Vu = Vu, or Vu = 0. This concludes the 
proof. □ 
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At this point we present the key lower bound in the one dimensional case. The main 
idea is to reduce the energy to an integral on the jump set of u and separate it into 
different parts, in order to show that the optimal curve has a structure similar to the 
one used for the construction in the upper bound and illustrated in Figure 5. For this 
we start from the reduction to intensity maps u which are characteristic functions, done 
in Lemma 4.2, and the truncation of u, discussed in Lemma 4.7. Then we show that 
we can consider also functions v which take only two values, and use this to subdivide 
7 into two subsets, a strategy similar to the one used in [16, Lemma 4.5] to obtain the 
relaxation of a line-tension model for dislocations in crystals. 

Proposition 4.8 (Lower bound for d = 1). Let d = 1, ^u~,v'^^v~ G M, n G he 

such that |[ii]| < 2^. Then K{u'^,u~,v~^,v~,n) > nim^+^j ^2 where 

K (A^^) = (a« |[rt]| + a-u |[u]|) (|A^’'’| + |n - Tap |M| ■ w~^ + {n - N~^) • r(;“| . 

Remark. If {w~ ■ n) • n) < 0 then the minimizer iV+ belongs to the segment [0, n] 
and makes the last term vanish, the bound reduces to iF > K {N +) = (a« IMI + Oi, I [u] I). 

Proof. By Lemma 4.1 we can assume without loss of generality that v~ < v~^. Let 
{u,v) G . By Lemma 4.7 we may further assume that v~ < v < v~^ . We write 
u = u'^ — [u\xq- for a set Q~ C Qn of finite perimeter. Using the abbreviation 7 = 
{dQ ~) n Qn and n"' for the outer normal to Q~ on 7 we have as in the proof of Lemma 
4.7 Du = [u\n'^'H^ L 7 and 


E[u,v,Qn] 



\Du\ + ay 



|Vu| dy + ap 



w ■ Du\ . 


We estimate 


E[u, v,Qn] > au\ [n] 77^ ( 7 ) -hai, 

/ Vv dy 


/ Vv dy 

+aF IMI 

[ w ■ n'^ dV.^ 


Qn 


jQt 




This is the key estimate to establish the lower bound, and it is sharp if the three inte¬ 
grands have a constant orientation on the respective domains. 

We observe that the quantity on the right-hand side depends only on the value of v 
on 7 . Indeed, since v is periodic in the direction orthogonal to n and = n, 


[ 'Vvdy= [ vn'^ dV.^ — V n= [{v — v )n'^ dV.^ 


JQ„ Jj J'y 

and the same on We therefore define G : L°°( 7 , [u“, u"*"]) 


by 


G[v\ = Uy 

f {v — V )n'^ dTL^ 

+ Oiy 

[{v — v~^)n'^ dTL^ ' 

+ o:p\[u\\ 

no + vnj dTL^ 


J'y 


J'y 


J'y 


so that (we recall that no is the time component, ni the space component) 

E[u,v,Qn] > an|[n]| 77 ^( 7 )-7 minG. 
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Existence of a minimizer in follows immediately from the convexity 

of the functional G. Even more, the functional G is weakly continuous on and 

therefore for any e > 0 there is u* G L°°{'y,{v~,v~^}) such that G(u*) < minG + e. This 
permits us to reduce to the situation where also the velocity v takes only two values. 
We split 7 into two subsets depending on the value of u*: 

7 “ = {?/G 7 ; u*(y) = u“} and 7 += {y G 7 : u*( 2 /) = u+} , 

and define 

N~ = f n'^ dV.^ and = f rG dV.^. 

J'y~ 

These quantities characterize the effective orientation of and 7 “, corresponding to 
the two normals in Proposition 4.6 and Lemma 4.5. Since u* is constant in each of the 
two subsets of 7 , by convexity one could assume that rG is also constant on each of 
them, and then rearrange so that 7 is composed of two segments. This illustrates the 
idea behind the energy bound, however it is not needed to conclude the proof. 

It suffices indeed to separate the integrals in the definition of G into a 7 ^ and a 7 “ 
part, then a short computation gives 

G[v^] = a^|[u]|(|iV+| + \N~\) + cTFlMlIno + | 

= a^|[u]|(|iV+| + \N~\) + aF\[u\\\w~^ ■ N~^ + w~ ■ N~\ , 

where = (l,u=*"). Since ^^( 7 ) > + |A^“| and by periodicity + N 

gives 

E[u,v,Qn]>K(N+)-e> min K(N+)-e 
for all («, v) G and e > 0, which concludes the proof. 

4.3 Proofs of Theorem 2.1 and 2.2 

At first we prove a lower bound in the higher dimensional case. 

Proposition 4.9 (Lower bound for d > 1). For {u,v) G let us define Q~ = 

{y ^ Qn ■ u{y) = u~} and W = {W)i=i d = fj v0 n dW^ G with v denoting 

j=o,..’,d 

the outer normal of Q~ on Ju- Then 

E[u,v,Qn] > aF\[u]\no + au\[u] \ + f(W), 
where /(IT) = |[tt]| Y,k=i ^kk + (|^^“ (g) n - IT| + |u+ (g) n - 1T|) . 


= n this 


□ 
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Proof. Repeating the first steps in the proof of Proposition 4.8 one verihes 


E[u,v,Qn] > aF\[u]\ / w ■ u + Uv / \Vv\ dy + (J„) 


' Qn 


u dPL'^ + q;„ 

/ Vv dy 

+ at. 

/ Vv dy 


Qn 


jQt 


+ Ctu I [u] 


I Ml ^no + +ay{\v (g) n - fR| + |r;+ (g) n - W\) + |MI 


= ap 

= ap IM Mo + an IMI + fiW) 


Since we do not have a sharp truncation result in this case the following steps in the 
proof of Proposition 4.8 based on the functional G do not extend to this situation. □ 

Now we proceed with the proofs of the main theorems. 

Proof of Theorem 2.1. First we treat the case (i), where the optical flow constraint can 
be satished locally. 

To prove the upper bound we first choose t G [0,1] such that — = 0. 

The upper bound follows then from Lemma 4.4(ii), taking a = v~t + u'’'(l — t). For the 
sake of illustration we give also an explicit construction here. Let (^ : M —)• M be a smooth 
monotone function such that t{—\) = 0, (^(0) = t and ^{\) = 1. We set 

u{y) =u~ + - u~)xyn <0 and v{y) = v~ + (u+ - v~)ip{y ■ n) (4.3) 

and observe that {u,v) G . A simple computation shows that AT < a^j |MI + a^ IMI- 
To prove the lower bound we choose {u, v) G A^ and verify using convexity and 
periodicity 

E[u, v,Qn] >au\ M I + a^, I [u] (g) n| = | M I + at, | [u] | . 

This concludes the proof of (i). 

The upper bound in (ii) follows immediately from the construction in Proposition 4.6. 
The lower bound in (ii) was proven in Proposition 4.8. 

Finally, to prove (iii) we observe that Lemma 4.5 shows that if n = Y^j=i ^ *5'^ 
then 

^ ^ ^ j\jj 

iF(n+,u“,u+,u“,n) = /i(^W) < = '^\N^\K{u+,u~,v+,v~,. 

j=i j=i j=i I I 

Using the bound from Lemma 4.4(ii) in each term in the sum gives the assertion. □ 

Proof of Theorem 2.2. (i): The first assertion immediately follows from Theorem 2.1 (i) 
and its proof. 

(ii): By Proposition 4.8 we have K > min,A(M^), where 

K{N^) = (a„ IMI + at, |M|) (|iV^| + |n - 1V+|) + ap |MI \w~ ■ n + . 
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We first show that if [u](re“ • n)ni > 0 then = 0 is a minimizer of Since 
N'^ I—7- |n — is convex, it lies above its tangent, |n — > 1 — n • . Using 

\w~ ■ n + > \w~ ■ n\ + sign(zt;“ • n)[r;]A^]^ for the last term we obtain 

K{N^) - K{0) > {au IMI + oiv I HI) {\N~^\ - n ■ N^) + IMI sign(rc“ • n)HiV+ . 


Using the fact that [r;](tc • n)ni is positive we obtain 
K{N+) - K{0) 


where 


On H + aj\v\ 




> \N+\ - n • iV++ eniAr+, 




(a„|H]| + ai;|HI)Hi| 

Since by assumption ^ G [0,2] we have 


n- N+ - CniN^ = (1 - + noiVo^ < |n| |A^+| = |A^+| , 


therefore K_{N^) > i^(0). 

Recalling the upper bound we have i^(0) < AT < -R(O), and since K{0) = i^(0) we 
conclude that in the case [u](rc“ • n)ni > 0 one has K = K{0) = |[ii]| + «« |H| + 

ap |[u]| H“ • n|, with a simple profile since we have set N~^ = 0 in the construction of 
the upper bound. The same result still holds if [u] = 0, as a simple inspection of the 
lower bound shows. 

Swapping (u+, u+) with {u~, v~) and using Lemma 4.1 we obtain that if [u](rc+-n)ni < 
0 then K = K(n) = «„ | HI + Ib]l + IHll H"*" ' ^1 > again with a simple profile. For 
[u] = 0 the two assertions coincide. Since for ni = 0 our assumption .^ < 2 is only satisfied 
in the trivial case [m] = 0 and [u] = 0, we have shown that if • n and w~ • n are nonzero 
and have the same sign the profile is simple and the interfacial energy is either iF(0) or 
iL(n); since they are both admissible the energy necessarily is the minimum of the two. 

(iii); At first we observe that Lemma 4.4 (ii) implies that ai?min{H“ • n|, \w~^ ■ n\} + 
Q!„| [u] I + a^l [«] I is an upper bound. Swapping n with —n we can assume that w~^ ■ n and 
w~ ■ n are strictly positive, swapping with {u~,v~) we can assume that 


0 < w ■ n < ■ n. 


From Proposition 4.9 we know that E[u, v, Qn] > ai? | HjHo + «« I Ml + with 

d 

f{W) = OF INI X/ (g) n - VU| + (g) n - 1U|) . 

k=l 


Notice that 


d 

f {v~ (g) n) = IMI X] vj:Uk + a^l [u] | . 

k=l 
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As w~^ ■ n > w~ • n > 0 it suffices to show that / has a minimum at v~ (8> n. Since / is 
convex this is the case if and only if 0 is a subgradient at v~ (8> n. It can be easily seen 
that the set of subgradients of / at (8) n is 

df {v~ (g) n) = OF I Ml {Sij)i<i<d, + cty (bi{0) - • 

0<j<d V \M\ J 


Hence 0 £ df (v (g) n) if and only if 


ir II /s: \ [u] (g) n 

Mil {^ij)l<i<d, rrm £ C 

Oiv 0<j<d I I 


(we recall that we are using the Euclidean norm on Squaring the left hand side 

shows that this is equivalent to 


f Olp\‘^ ,, , CKFir II ■ Td, 


which in turn is the same as 


ir 1 , , \w\ ■ n 


This holds since we are in the case \w\-n > 0 and by assumption 2 q;^| M]-n| > [u]| | [u]|. 

We finally prove (iv). The upper bound follows from Theorem 2.1(iii) with 1 = 1. To 
prove the lower bound, we observe that the construction in Lemma 4.2 does not modify 
the property of being one-dimensional, hence we can assume that u,v £ are of the 
form u = u~ + [u]xw{y • n), v = v{y ■ n), for some set of finite perimeter uj C and 
some function u : M —)■ W^. Then, setting U £ Ju = dio, 


ri/2 

E[u,v,Qn]>au\[u]\+ay / \v'\{t)dt + aF\[u]\\{l,v{E)) ■ n 

4 - 1/2 


With a = u(t*) the assertion follows. 


□ 


4.4 Illustration of expected microstructures for d > 1 

In the higher dimensional case (d > 1) the upper bound in Theorem 2.1 (iii) refers to 
a set of vectors {N^,... ,N^). In the case d = 1 the corresponding vectors and N~ 
explicitly appear in a geometric construction of a microscopic pattern in the proof of 
Proposition 4.6. Using the convexity of the microscopic energy K stated in Lemma 4.5 
this explicit construction is not required in the proof of Theorem 2.1 (iii). Nevertheless, 
explicit microscopic patterns can be constructed, where the vectors W are weighted 
normals with being the normal on a set of interface facets of total area |W |. Figure 
7 sketches for d = 2 such a microscopic pattern with three weighted normals 
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Figure 7: Sketch of a microscopic pattern corresponding to an explicit construction of 
the upper bound of K for d = 2 (left: macroscopic configuration, middle: 
microscopic pattern, right: top view of the pattern) 


where Q is a suitable rotation in The resulting effective interface normal n = 
= Q(0,0,1)^ is the macroscopic normal on the jump set The 
underlying pattern is based on nested lamination, i.e. a lamination pattern of facets 
perpendicular to and (plotted in light and dark orange) is altered with facets 
perpendicular to (plotted in green). To compensate the lack of rank-1 consistency a 
thin transition pattern is introduced in between (plotted in grey). 
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